Data processing system and method

ABSTRACT

A data processing system includes a data processing arrangement, wherein the data processing arrangement includes computing hardware for executing one or more software products, wherein execution of the one or more software products configures the data processing arrangement to access data from a file system arrangement. The data processing arrangement is operable to load a dynamic linker that is operable to include an intercept library that intercepts file access operations of an executable software product wherein: (i) a virtual file that does not exist on the file system arrangement becomes accessible to the executable software product; (ii) the virtual file is a result of either (a) a just-in-time translation of one or more real files of the file system arrangement, wherein the just-in-time translation is made from one file format to another file format, or (b) conversion of the access operation to an equivalent cloud object-storage access operation(s) on a real object located on the cloud object-storage, or (c) a combination of the just-in-time translation of (a) with the object access conversion from (b). Optionally, at least one of: (iii) the virtual file is a different file format to the real file/object, wherein a compressed file format and the virtual file are mutually different file formats; and (iv) the file formats are genomic file formats.

TECHNICAL FIELD

The present disclosure relates to data processing systems. Moreover, thepresent disclosure concerns methods of using aforesaid data processingsystems for processing data, for example for processing genomics data.Yet additionally, the present disclosure is concerned with computerprogram products comprising non-transitory computer-readable storagemedia having computer-readable instructions stored thereon, thecomputer-readable instructions being executable by a computerized devicecomprising processing hardware to execute the aforesaid methods.

BACKGROUND

Contemporary data processing systems are often required to access datastored in one or more repositories when performing data processingfunctions; the repositories of data can include, for example, genomicdata. The data processing systems are operable to function undersoftware control, by executing one or more software products.

In generating software products, it is contemporary practice to employ alinker to compile various items of software to generate executablesoftware code. The executable software code is known as a softwareproduct. Moreover, various types of linkers are known, for example asdescribed in Wikipedia, wherein a dynamic linker can be influenced intomodifying its behaviour during either a given program's execution orlinking of the given program, and examples of a dynamic linker can befound described in run-time linker manual pages for various Unix-likesystems; UNIX® is a trademark. A typical modification of a behaviour ofsuch a dynamic linker is provided in:

LD_LIBRARY_PATH and LD_PRELOAD environment variables,

wherein the dynamic linker adjusts a runtime linking process bysearching for shared libraries at alternate locations and by forciblyloading and linking libraries that would otherwise not be loaded andlinked, respectively. An example of modifying an executable behaviourvia a dynamic linker is zlibc, also known as “uncompress.so”, whichfacilitates transparent decompression when used through the LD_PRELOADhack; as a result, it is possible to read a pre-compressed (gzipped)file data on BSD and Linux® systems as if the files were not compressed,essentially allowing a given user to add transparent decompression to anunderlying file system, although with some caveats. In order to achievesuch a functionality, a mechanism is employed that is flexible, allowingtrivial adaptation of a same given code to perform additional oralternate processing of data during a given file read, prior to aprovision of data to a given user process that has requested it.

However, contemporary known dynamic linkers do not perform adequatelywell in data processing systems to allow a wide variety of data inrepositories to be modified in a dynamic manner.

SUMMARY

The present disclosure seeks to provide an improved data processingsystem that is able to translate file accesses, for example tocompressed genomic data or to cloud object storage, in a more flexibleand dynamic manner.

In a first aspect, there is provided a data processing system includinga data processing arrangement, wherein the data processing arrangementincludes computing hardware for executing one or more software products,wherein execution of the one or more software products configures thedata processing arrangement to access data from a file systemarrangement,

characterized in that the data processing arrangement is operable toload a dynamic linker that is operable to include an intercept librarythat intercepts file access operations of an executable software productwherein:

-   (i) a virtual file that does not exist on the file system    arrangement becomes accessible to the executable software product;-   (ii) the virtual file is a result of either (a) a just-in-time    translation of one or more real files of the file system    arrangement, wherein the just-in-time translation is made from one    file format to another file format, or (b) conversion of the access    operation to an equivalent cloud object-storage access operation(s)    on a real object located on the cloud object-storage, or (c) a    combination of the just-in-time translation of (a) with the object    access conversion from (b).

Optionally, in the data processing system, at least one of:

-   (iii) the virtual file is a different file format to the real    file/object, wherein a compressed file format and the virtual file    are mutually different file formats; and-   (iv) the file formats are genomic file formats.

The invention is of advantage in that the dynamic linker, via use of theintercept library, is operable to cause the data processing system to bemore versatile in utilizing dynamically changing data files and datafile formats when performing computations, for example computations uponacquired data from sensor arrangements, for example associated withgenomic readout.

In a second aspect, there is provided a method of using a dataprocessing system including a data processing arrangement, wherein thedata processing arrangement includes computing hardware for executingone or more software products, wherein execution of the one or moresoftware products configures the data processing arrangement to accessdata from a file system arrangement,

characterized in that the method includes operating the data processingarrangement to load a dynamic linker that is operable to include anintercept library that intercepts file access operations of anexecutable software product wherein:

-   (i) a virtual file that does not exist on the file system    arrangement becomes accessible to the executable software product;-   (ii) the virtual file is a result of either (a) a just-in-time    translation of one or more real files of the file system    arrangement, wherein the just-in-time translation is made from one    file format to another file format, or (b) conversion of the access    operation to an equivalent cloud object-storage access operation(s)    on a real object located on the cloud object-storage, or (c) a    combination of the just-in-time translation of (a) with the object    access conversion from (b).

Optionally, in the method, at least one of:

-   (iii) the virtual file is a different file format to the real    file/object, wherein a compressed file format and the virtual file    are mutually different file formats; and-   (iv) the file formats are genomic file formats.

Optionally, in the data processing system, at least one of:

-   (iii) the virtual file is a different file format to the real    file/object, wherein a compressed file format and the virtual file    are mutually different file formats; and-   (iv) the file formats are genomic file formats.

In a third aspect, embodiments of the present disclosure provide acomputer program product comprising a non-transitory computer-readablestorage medium having computer-readable instructions stored thereon, thecomputer-readable instructions being executable by a computerized devicecomprising processing hardware to execute a method pursuant to theaforementioned second aspect.

Additional aspects, advantages, features and objects of the presentdisclosure would be made apparent from the drawings and the detaileddescription of the illustrative embodiments construed in conjunctionwith the appended claims that follow.

It will be appreciated that features of the present disclosure aresusceptible to being combined in various combinations without departingfrom the scope of the present disclosure as defined by the appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The summary above, as well as the following detailed description ofillustrative embodiments, is better understood when read in conjunctionwith the appended drawings. For the purpose of illustrating the presentdisclosure, exemplary constructions of the disclosure are shown in thedrawings. However, the present disclosure is not limited to specificmethods and apparatus disclosed herein. Moreover, those in the art willunderstand that the drawings are not to scale. Wherever possible, likeelements have been indicated by identical numbers.

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the following diagram wherein:

FIG. 1 is a schematic illustration of a data processing system pursuantto the present disclosure; and

FIG. 2 is an illustration of a flow diagram of a method of (for)operating the data processing system of FIG. 1 .

In the accompanying diagrams, an underlined number is employed torepresent an item over which the underlined number is positioned or anitem to which the underlined number is adjacent.

DETAILED DESCRIPTION OF EMBODIMENTS

The following detailed description illustrates embodiments of thepresent disclosure and ways in which they can be implemented. Althoughsome modes of carrying out the present disclosure have been disclosed,those skilled in the art would recognize that other embodiments forcarrying out or practising the present disclosure are also possible.

A virtual file is a file (or directory) that is not actually present ona computer system's file system. Whereas a virtual file system maycomprise an entire file system tree of virtual files, a virtual file mayadditionally appear to be located at a path within an existing real filesystem, even though the virtual file is not actually present on the realfile system.

File systems, including virtual file systems, are required byPOSIX-compliant Operating Systems to be mounted at a path, usually byprivileged users. In contrast, a FUSE system in Linux® allows anon-privileged user to mount file systems, including virtual filesystems, at a path with the assistance of the FUSE kernel model.However, this approach may not be accessible within tighter securityprivileges of a container-environment such as Docker®, without explicitrelaxation of these restrictions. When one user mounts such a filesystem, its presence is also generally visible to all other users, whichmay not be desirable.

In some circumstances, it is useful to be able to make a virtual filesystem available that either (a) works without a system mount point, (b)works in-place within the existing mounted file systems, and/or (c) canwork with restricted privileges such as found in restricted environmentslike Docker®.

An alternative approach to using mounted virtual file systems, is to useDynamic Linking to intercept and modify accesses by an executable to thefile system. Such a mechanism includes LD_PRELOAD in Linux-based systemsand DLYD_INSERT_LIBRARIES for BSD-based systems including Mac OS. Here,upon loading the executable, resolving of symbols to dynamic librariessuch as glibc.so used by most Linux executables to access file systems,can be intercepted by an alternate supplied library, providing matchingsymbol names. The alternate library can intercept calls by theexecutable to the matching symbols, so as to alter the functionality ofthose symbols, which in turn alters the overall behaviour of theexecutable. For example, an ‘open64’ symbol is assumed by an executableto allow opening of a file for subsequent accesses, and returns afile-handle. The alternate library can intercept calls to this symbol soas to provide alternate behaviour, such as opening a different file fromthe one initially specified. This interception mechanism for file-systemaccesses has the advantage of not requiring any special privileges, andenables virtual files to appear to reside within existing mountedfile-systems. However, such an interception mechanism for file-systemaccesses has a number of shortcomings that has not made it practical forgeneral use.

The shortcomings are:

1. Although file system accesses can readily be intercepted by hookinginto operations such as open, read and write, memory-mapped files havetheir read/write operations handled by the operating system whichdirectly reads/writes data to the file-system through the kernel, thusnot allowing interception by the library.

2. Statically-compiled binaries, or binaries which do not use librariessuch as glibc.so to access the file system, but instead directly invokeoperating system calls, cannot be intercepted by this Dynamic Linkingmechanism.

In overview, referring to FIG. 1 , there is provided a data processingsystem 10 a data processing arrangement 20, wherein the data processingarrangement 20 includes computing hardware 30 for executing one or moresoftware products 40, wherein execution of the one or more softwareproducts 40 configures the data processing arrangement to access datafrom a file system arrangement, characterized in that the dataprocessing arrangement 10 is operable to load a dynamic linker 50 thatis operable to include an intercept library 50 that intercepts fileaccess operations of an executable software product wherein:

-   (i) a virtual file that does not exist on the file system    arrangement becomes accessible to the executable software product;-   (ii) the virtual file is a result of either (a) a just-in-time    translation of one or more real files of the file system    arrangement, wherein the just-in-time translation is made from one    file format to another file format, or (b) conversion of the access    operation to an equivalent cloud object-storage access operation(s)    on a real object located on the cloud object-storage, or (c) a    combination of the just-in-time translation of (a) with the object    access conversion from (b).

Optionally, in the data processing system, at least one of:

-   (iii) the virtual file is a different file format to the real    file/object, wherein a compressed file format and the virtual file    are mutually different file formats; and-   (iv) the file formats are genomic file formats.

Optionally, the compressed file format is a compressed genomic fileformat, and another file format is another genomic file format.

The bulk of genomic data is already compressed in standard formats aseither BAM or FASTQ.gz files, utilising ZLIB/GZIP-based compressiontechnology. As individual files can be many GBs (Gigabytes) andsometimes TBs (Terabytes) in size, organisations have the challenge ofstoring and transferring these files. Better compression would beadvantageous in reducing the cost of storage and the time for datatransfers. However, compressing to a new or updated file formatnecessarily breaks compatibility with the existing ecosystem ofbioinformatics tools and pipelines that do not support this new fileformat. While it is possible to decompress back to the original fileformat (such as BAM or FASTQ.gz) and then feed these into the tools andpipelines, this needs to be specified explicitly as part of thepipeline, or time must be taken to first decompress the entire filebefore processing, even when only a small portion is actually used forthe analysis. For this reason, it is desirable to be able totransparently convert from the new better-compressed file format, to theless-compressed, but widely compatible file format.

Another issue with processing genomic data is that pipelines arecomposed of executables that can read/write storage via POSIX fileaccesses, and are not built for reading or writing to cloud storagewhich usually operate via REST accesses. It would be advantageous ifthese tools/pipelines could access cloud storage as though they wereregular files.

Optionally, in the data processing system, the dynamic linker includesthe intercept library in a forceful manner.

Optionally, in the data processing system, the executable softwareproduct is operable to access genomic data via use of a ptrace call,wherein the ptrace call allows for manipulation of file descriptors,data memory and data registers. More optionally, in the data processingsystem, the ptrace call is operable to intercept forcibly calls made bythe data processing system of a child process that is executable byintercepting file system calls via a kernel-provided executable tracingfunction wherein:

-   (i) a virtual file that does not exist on the file system    arrangement is made accessible to the data processing system;-   (ii) the virtual file is a just-in-time translation of one or more    files on the file system arrangement from one file format to another    file format;-   (iii) the real file is a compressed genomic file format and the    virtual file is another genomic file format; and-   (iv)(*) in operation, a system call for opening the virtual file is    intercepted, and firstly handled by ensuring that a virtual file    system mount is available, and redirecting the system call to a file    on the virtual file system instead.

Optionally, the data processing system is operable to intercept filesystem accesses of a child process so as to provide access to virtualfiles to the child process, wherein the data processing system isoperable to:

(i) intercept a library using a Dynamic Linking interception mechanismsuch as LD_PRELOAD;

(ii) intercept calls in the library for creating a new child process(for example, by intercepting exec variants, such as execve, andfork/vfork in Linux);

(iii) examine the executable of the new child process to determinewhether or not it has dependencies on an appropriate intercept library(for example, by checking if it has a dynamic dependency on glibc.so),

and wherein:

(iv) if the child process has dependencies on the intercept library,then allowing the child process to be created, but ensuring that theDynamic Library interception mechanism is enabled (for example, byensuring that the LD_PRELOAD environment variable includes the requisiteinterception library), wherein the requisite intercept libraryintercepts file-system accesses so as to provide the virtual files; or

(v) if the child executable does not have dependencies on the interceptlibrary, then checking to see whether an alternate interceptionmechanism is available (i.e. whether the parent process has sufficientprivileges to apply the alternate interception method), wherein if thealternate interception mechanism is available, then applying thealternate interception mechanism to the child process.

Optionally, if multiple such alternate interception mechanisms areavailable, then the data processing system is operable to choose onesuch available alternate interception mechanism.

Optionally, the alternate interception mechanism allows the childprocess to proceed, but sets up interception system calls of the childprocess (for example, by using the ptrace mechanism in Linux). Moreoptionally, for performance, the interception of system calls arerestricted to only those of interest (for example, by using seccompfilters on Linux). In this manner, all file-system operations can beintercepted such that any operations on virtual files are translated.

Optionally, the alternate interception mechanism is to utilise aJust-In-Time recompiler (such as Intel's PIN tool or HP-MIT's DynamoRIO)operating on the child process, wherein the Just-In-Time recompiler isconfigured to detect and intercept system calls by the child process. Inthis manner, system calls that access the file-system can be redirectedto alternative code that provides the virtual files.

Optionally, the data processing system is operable to redirect anyfilename-based (or pathname-based) access to a virtual file, to anequivalent entry within a virtual file system mount point (such as, aFUSE mount to a temporary restricted access directory for Linux),wherein if no such mount point is currently present, a new mount iscreated Just-In-Time prior to the modified system call being invoked. Inthis scheme, the system call for operations such as ‘open file’ wouldreturn a valid file handle, wherein the read/write operations on thatfile handle would not need further system-call interception, thusimproving performance, but would instead be intercepted via themount-based Virtual File System method.

More optionally, in the data processing system, in (iv)(*), the virtualfile system mount is implemented as a temporary directory, wherein, ifthe mount point is not already present, the data processing system isoperable automatically to create the mount point with the virtual filesystem so that it is present.

Optionally, in the data processing system, the dynamic linker isoperable to intercept the system calls of a child process executable byjust-in-time recompiliation of sections of binary code just beforebinary code is run, wherein:

-   (i) the virtual file that does not exist on the file system    arrangement is made accessible;-   (ii) the virtual file is a just-in-time translation of one or more    real files on the file system arrangement, wherein the just-in-time    translation is made from one file format to another file format-   (iii) the real file is a compressed genomic file format and the    virtual file is another genomic file format; and-   (iv) a system call for opening the virtual file is intercepted, and    handled by firstly ensuring that a virtual file system mount is    available, and then redirecting the system call to a file on the    virtual file system instead.

More optionally, in data processing system, the virtual file systemmount is implemented as a temporary directory, wherein there isautomatically created a mount point with the virtual file system so thatit is present.

Optionally, in the data processing system, the just-in-time translationof a transparent access of genomic data is operable to combine contentfrom multiple genomics files and present it as one genomic file, for anyone or combination of:

-   (i) wherein the merged content is quality score data;-   (ii) wherein the merged content is read name information;-   (iii) wherein the merged content is auxiliary tags of mapped genomic    reads;-   (iv) wherein the merged content consists of separate genomic    regions;-   (v) wherein the merged content consists of multiple genomic    samples/specimens; and-   (vi) wherein the separate genomic files represent different regions,    samples or other separable parts of a given genome.

Optionally, in the data processing system, the dynamic linker isforcibly loaded and employs in operation an intercept library thatintercepts file access operations of an executable software productwherein:

-   (i) the creation of a new child process preserves an intercept    library in an associated interception environment variable.

Optionally, in the data processing system, the dynamic linker isoperable to employ an intercept library that intercepts file accessoperations of an executable software product wherein:

-   (i) the intercept library detects if a program is being submitted to    a job submission system, and if so:-   (ii) a temporary shell script is created that preserves an    interception environment variable prior to calling an original    program; and-   (iii) there is submitted the new temporary script to the job    submission system, instead of the original program

More optionally, in the data processing system, before implementing(iii), the data processing system is operable to:

-   (iv) detect if the original program is a script containing    job-submission-system specific metadata, and if so copy this    metadata information to a new temporary shell script.

Optionally, the data processing system is operable to providetransparent access of genomic data, such that accesses under a virtualpath (for example, /pgs3/) are redirected to a cloud storage arrangementby translating operations to equivalent translated requests that aresent over the Internet to a cloud storage arrangement provider.

There are also compound concerns, where accessing genomic data on cloudstorage presents challenges not just in the ability for tools andpipelines to directly stream access from cloud storage, but also suffersconsiderable delays and costs due to the bulkiness of the genomic data.

It would be advantageous if the objects in cloud storage utilisedbetter-compressed file formats, and are Just-In-Time translated to thestandard file format. In such a scheme, the file-system accesses to thestandard file format are converted into equivalent cloud accesses to thebetter-compressed file format. Due to the better compression, less datawould then be needed to be transferred from the cloud object storage,thus accelerating the access, at the cost of a computational overheadfor translating the better-compressed data to the form of the standardfile format.

More optionally, in the data processing system:

-   (i) a corresponding virtual file that does not exist on the cloud    storage arrangement is accessible to the data processing system;-   (ii) the virtual file is a just-in-time translation of one or more    corresponding real objects on the cloud storage arrangement from one    file format to another file format;-   (iii) the real object is a compressed genomic file format and the    virtual file is another genomic file format.

Optionally, the data processing system is operable to providetransparent access of genomic data, such that dynamic linker is operableto provide the intercept library for memory-mapped file accessoperations of an executable to a virtual file by:

-   (i) registering a page fault interrupt handler;-   (ii) creating a virtual region of size as requested by a    memory-mapped file mapping operation, but protected to prevent reads    and writes;-   (iii) on read access of one or more given protected pages, replacing    the one or more pages with corresponding translated content from the    real file, and allowing the one or more given protected pages to be    accessible for reads and/or writes; and-   (iv) maintaining a list of the one or more pages of translated    content, and upon hitting a memory consumption limit free up memory    taken by translated content, choosing one or more pages of the    translated content, freeing memory of the one or more pages and    protecting these page regions again to prevent further reads and    writes; and-   (v) wherein a choice of which page to free is made by employing a    LRU (least recently used), LFU (least frequently used) or other    replacement heuristic.

Optionally, the data processing system is operable to providetransparent access of genomic data such that it intercepts the dynamiclinker and is operable to provide the intercept library to enablememory-mapped file access operations of an executable to a virtual file,wherein:

-   (i) a system call for memory-mapping the virtual file is    intercepted, and handled firstly by ensuring that a virtual file    system mount is available (perhaps in a temporary directory, here    ‘ensure’ means that if the mount point is not already present, to    automatically create said mount point with virtual file system so    that it is present), and then redirecting the memory-mapping    operation to a file on the virtual file system instead.

According to another aspect, there is provided a method of using a dataprocessing system 10 including a data processing arrangement 20, whereinthe data processing arrangement 20 includes computing hardware 30 forexecuting one or more software products 40, wherein execution of the oneor more software products 40 configures the data processing arrangement20 to access data from a file system arrangement,

characterized in that the method includes operating the data processingarrangement to load a dynamic linker that is operable to include anintercept library that intercepts file access operations of anexecutable software product wherein:

-   (i) a virtual file that does not exist on the file system    arrangement becomes accessible to the executable software product;-   (ii) the virtual file is a result of either (a) a just-in-time    translation of one or more real files of the file system    arrangement, wherein the just-in-time translation is made from one    file format to another file format, or (b) conversion of the access    operation to an equivalent cloud object-storage access operation(s)    on a real object located on the cloud object-storage, or (c) a    combination of the just-in-time translation of (a) with the object    access conversion from (b).

Optionally, in the method, at least one of:

-   (iii) the virtual file is a different file format to the real    file/object, wherein a compressed file format and the virtual file    are mutually different file formats; and-   (iv) the file formats are genomic file formats.

Optionally, both (iii) and (iv) apply in a given method of the presentdisclosure.

Optionally, the compressed file format is a compressed genomic fileformat, and another file format is another genomic file format.

Referring to FIG. 2 , there is shown a slow chart of steps of a methodthat is implemented using the data processing system 10 of FIG. 1 .

The method includes a first step 200 of providing a data processingsystem including a data processing arrangement, wherein the dataprocessing arrangement includes computing hardware for executing one ormore software products, wherein execution of the one or more softwareproducts configures the data processing arrangement to access data froma file system arrangement.

The method further includes a second step 210 of operating the dataprocessing arrangement 10 to load a dynamic linker that includes anintercept library that intercepts file access operations of anexecutable software product wherein:

-   (i) a virtual file that does not exist on the file system    arrangement becomes accessible to the executable software product;-   (ii) the virtual file is a result of either (a) a just-in-time    translation of one or more real files of the file system    arrangement, wherein the just-in-time translation is made from one    file format to another file format, or (b) conversion of the access    operation to an equivalent cloud object-storage access operation(s)    on a real object located on the cloud object-storage, or (c) a    combination of the just-in-time translation of (a) with the object    access conversion from (b).

Optionally, in the method of FIG. 2 , at least one of:

-   (iii) the virtual file is a different file format to the real    file/object, wherein a compressed file format and the virtual file    are mutually different file formats; and-   (iv) the file formats are genomic file formats.

Optionally, both (iii) and (iv) pertain to the method of FIG. 2 .

According to another aspect there is provided a computer program productcomprising a non-transitory computer-readable storage medium havingcomputer-readable instructions stored thereon, the computer-readableinstructions being executable by a computerized device comprisingprocessing hardware to execute the aforementioned method.

Modifications to embodiments of the present disclosure described in theforegoing are possible without departing from the scope of the presentdisclosure as defined by the accompanying claims. Expressions such as“including”, “comprising”, “incorporating”, “consisting of”, “have”,“is” used to describe and claim the present invention are intended to beconstrued in a non-exclusive manner, namely allowing for items,components or elements not explicitly described also to be present.Reference to the singular is also to be construed to relate to theplural. Numerals included within parentheses in the accompanying claimsare intended to assist understanding of the claims and should not beconstrued in any way to limit subject matter claimed by these claims.

The phrases “in an embodiment”, “according to an embodiment” and thelike generally mean the particular feature, structure, or characteristicfollowing the phrase is included in at least one embodiment of thepresent disclosure, and may be included in more than one embodiment ofthe present disclosure. Importantly, such phrases do not necessarilyrefer to the same embodiment.

APPENDIX (INVENTOR'S NOTES)

A Transparent Access Layer

From Wikipedia entry on Dynamic Linker:

“The dynamic linker can be influenced into modifying its behaviourduring either the program's execution or the program's linking, and theexamples of this can be seen in the run-time linker manual pages forvarious Unix-like systems. A typical modification of this behaviour isthe use of:

LD_LIBRARY_PATH and LD_PRELOAD environment variables,

which adjust the runtime linking process by searching for sharedlibraries at alternate locations and by forcibly loading and linkinglibraries that would otherwise not be, respectively. An example isZlibc®, also known as “uncompress.so”, which facilitates transparentdecompression when used through the LD_PRELOAD hack; as a result, it ispossible to read pre-compressed (gzipped) file data on BSD and Linux®systems as if the files were not compressed, essentially allowing a userto add transparent compression to the underlying filesystem, althoughwith some caveats. The mechanism is flexible, allowing trivialadaptation of the same code to perform additional or alternateprocessing of data during the file read, prior to the provision of saiddata to the user process that has requested it.”

Transparent Access of Genomic Data (Main Method)

Method:

Forcibly loading, via a Dynamic Linker, an intercept library thatintercepts the file access operations of an executable such that:

-   (i) a virtual file that does not exist on the file-system is    accessible;-   (ii) wherein the virtual file is a just-in-time translation of a    real file(s) on the file-system from one file format to another file    format; and-   (iii) wherein the real file is a compressed genomic file format and    the virtual file is another genomic file format.

Alternate Method 1 of Transparent Access of Genomic Data:

Using system-call interception, such as via ptrace.

From Wikipedia entry on ptrace:

“ptrace is used by debuggers (such as qdb and dbx), by tracing toolslike strace and Itrace, and by code coverage tools. ptrace is also usedby specialised programs to patch running programs, to avoid unfixed bugsor to overcome security features. It can further be used as a sandboxand as a runtime environment simulator (like emulating root access fornon-root software).

By attaching to another process using the ptrace call, a tool hasextensive control over the operation of its target. This includesmanipulation of its file descriptors, memory, and registers. It cansingle-step through the target's code, can observe and intercept systemcalls and their results, and can manipulate the target's signal handlersand both receive and send signals on its behalf. The ability to writeinto the target's memory allows not only its data store to be changed,but also the application's own code segment, allowing the controller toinstall breakpoints and patch the running code of the target.”

Prior art on this type of access method but that does not do translationor virtual file systems:

http://www.alfonsobeato.net/c/modifying-system-call-arguments-with-ptrace/

http://www.alfonsobeato.net/c/filter-and-modify-system-calls-with-seccomp-and -ptrace/

Method:

Forcibly intercepting the system calls of a (child process) executableby intercepting file system calls via kernel-provided executable tracingfunction such that:

-   (i) a virtual file that does not exist on the file-system is    accessible;-   (ii) wherein the virtual file is a just-in-time translation of a    real file(s) on the file-system from one file format to another file    format;-   (iii) wherein the real file is a compressed genomic file format and    the virtual file is another genomic file format; and-   (iv) wherein a system call for opening said virtual file is    intercepted, and handled by first ensuring that a virtual file    system mount is available (perhaps in a temporary directory, here    ‘ensure’ means that if the mount point is not already present, to    automatically create said mount point with virtual file system so    that it is present), and redirecting the system call to a file on    said virtual file system instead.

Alternate Method 2 of Transparent Access of Genomic Data:

This is based on instrumentation such as by Intel's PIN.

From Wikipedia entry on PIN:

“Pin performs instrumentation by taking control of the program justafter it loads into the memory. Then just-in-time recompiles (JIT) smallsections of the binary code using pin just before it is run. Newinstructions to perform analysis are added to the recompiled code. Thesenew instructions come from the Pintool. A large array of optimizationtechniques are used to obtain the lowest possible running time andmemory use overhead. As of June 2010, Pin's average base overhead is 30percent (without running a pintool).”

Method:

Forcibly intercepting the system calls of a (child process) executableby just-in-time recompiliation of sections of the binary code justbefore it is run, such that:

-   (i) a virtual file that does not exist on the file-system is    accessible;-   (ii) wherein the virtual file is a just-in-time translation of a    real file(s) on the file-system from one file format to another file    format;-   (iii) wherein the real file is a compressed genomic file format and    the virtual file is another genomic file format; and-   (iv) wherein a system call for opening said virtual file is    intercepted, and handled by first ensuring that a virtual file    system mount is available (perhaps in a temporary directory, here    ‘ensure’ means that if the mount point is not already present, to    automatically create said mount point with virtual file system so    that it is present), and redirecting the system call to a file on    said virtual file system instead.

Note that like the main method for interception, the Alternate methodscan also work by intercepting actual file read (write) and seekoperations to the virtual file rather than just the file open operation.However (iv) also allows a more efficient interception method so thatthe file-open operation is redirected to a path on a mounted virtualfile system (e.g. FUSE under Linux), and so that the actual file readand seek operations are intercepted at the mounted virtual file systemlayer, instead of via intercepting the corresponding system calls.

Decision Between the Two Methods of Accessing Genomic Data:

Method:

Wherein the executable being intercepted is examined to see if it has adynamic library dependency on an interceptable library (such as glibc)and if not, then an alternate access method is used instead of the mainmethod. The alternate method can itself be chosen based on whether theAlternate method 1 is available for use on the executable (i.e. if thereare sufficient user security privileges) and if not using the Alternatemethod 2 instead.

Transparent Access of Tiered/Separated Genomic Data

Method:

Wherein the just in time translation of the transparent access ofgenomic data (and alternates) can combine content from multiple genomicsfiles and present it as one genomic file, for any one or combination of:

-   (i) Wherein the merged content is quality score data;-   (ii) Wherein the merged content is read name information;-   (iii) Wherein the merged content is auxiliary tags of mapped genomic    reads;-   (iv) Wherein the merged content consists of separate genomic    regions; and-   (v) Wherein the merged content consists of multiple genomic    samples/specimens;

Wherein the just in time translation of the transparent access ofgenomic data (and alternates) can take one genomic file and present itas multiple genomic files:

-   (i) wherein the separate genomic files represent different regions,    samples or other separable parts of the genome.

Protect Interception Ability with Child Processes

In Linux, the LD_PRELOAD environment variable can be configured to loadan interception library, however this means that if the environmentvariable is modified by a process or if a child process is invokedwithout it, then the interception capability is lost. Similarenvironment variables exist in MacOS and BSD-based operating systems. Wewill call these interception environment variables.

Method:

Forcibly loading, via a Dynamic Linker, an intercept library thatintercepts the file access operations of an executable such that:

-   (i) the creation of a new (child) process preserves the intercept    library in the interception environment variable.

Protect Interception Ability with Job Submission Systems

Job submission systems (such as for HPC systems) may not preserveinterception environment variables for a job submission due to it beingconsidered a security risk.

Method:

Forcibly loading, via a Dynamic Linker, an intercept library thatintercepts the file access operations of an executable such that:

-   (i) it detects if a program is being submitted to a job submission    system, and if so;-   (ii) a temporary shell script is created that preserves the    interception environment variable prior to calling the original    program;-   (iii) optionally detecting if the program is a script containing    job-submission-system specific metadata, and if so copying this    metadata information to the new temporary shell script; and-   (iv) submitting the new temporary script to the job submission    system, instead of the original program.

Extensions for Cloud Storage

That enables access to/from cloud (e.g. AWS S3) storage as virtualfiles. Unlike prior art, this is via the above interception methods andwith a virtual file with different file format to the underlying objectfile format.

Method:

As per the transparent access of genomic data (and alternatives) suchthat accesses under a virtual path (e.g. /pgs3/) are redirected to cloudstorage by translating operations to equivalent translated requests thatare sent over the internet to the cloud storage provider and such that:

-   (i) a corresponding virtual file that does not exist on the cloud    storage is accessible;-   (ii) wherein the virtual file is a just-in-time translation of a    corresponding real object(s) on the cloud storage from one file    format to another file format; and-   (iii) wherein the real object is a compressed genomic file format    and the virtual file is another genomic file format.

For example, accesses to/pgs3/mybucket/myfile.bam are redirected to anobject that exists at cloud object location s3://mybucket/myfile.cram(with a different file format), and where a corresponding object ats3://nnybucket/myfile.bam does not exist.

Handling Memory Mapped Files

Unfortunately accesses to memory-mapped files are harder to handle inthe case of forcibly loading, via a Dynamic Linker, an intercept librarythat intercepts the file access operations of an executable. This isbecause such file accesses are meant to occur by merely accessing amemory location within the region of a memory-mapped file, rather thanas a call to an interceptable library function.

Method:

As per the Transparent access of genomic data (and alternates) such thatit intercepts the memory-mapped file access operations of an executableto a virtual file by:

-   (i) Registering a page fault interrupt handler;-   (ii) Creating a virtual region of size as requested by memory-mapped    file mapping operation, but protected to prevent reads and writes;-   (iii) On read access of a said protected page, replacing the page    (and optionally surrounding pages, or optionally prefetching    subsequent pages) with corresponding translated content from the    real file, and allowing said page(s) to be accessible for reads    (and/or writes);-   (iv) Maintaining a list of said pages of translated content, and    upon hitting a memory consumption limit free up memory taken by    translated content, choosing a page (or pages) of said translated    content, freeing memory of said page(s) and protecting these page    regions again to prevent further reads and writes; and-   (v) wherein the choice of which page to free is by a LRU (least    recently used), LFU (least frequently used) or other replacement    heuristic.

Alternate Method for Handling Memory Mapped Files

As per the Transparent access of genomic data (and alternates) such thatit intercepts the memory-mapped file access operations of an executableto a virtual file:

-   (i) wherein a system call for memory-mapping said virtual file is    intercepted, and handled by first ensuring that a virtual file    system mount is available (perhaps in a temporary directory, here    ‘ensure’ means that if the mount point is not already present, to    automatically create said mount point with virtual file system so    that it is present), and redirecting the memory-mapping operation to    a file on said virtual file system instead.

1-21. (canceled)
 22. A data processing system, comprising: a dataprocessing arrangement comprising computing hardware, the dataprocessing arrangement configured to access data from a file systemarrangement, wherein the data processing arrangement uses a dynamiclinker that uses an intercept library to intercept file accessoperations of an executable software product, wherein the interceptlibrary employs multiple interception mechanisms, including a firstinterception mechanism to intercept dynamic binaries and a secondinterception mechanism different from the first interception mechanism,and wherein the intercept library employs the first interceptionmechanism to intercept a launch by a parent process of a new childprocess associated with an executable binary; and a computer-readablestorage medium storing computer-readable instructions that, whenexecuted by the computing hardware, cause the data processingarrangement to perform: intercepting, using the intercept library, thelaunch of the new child process by the parent process; determiningwhether the executable binary has a dependence on an operating systemlibrary; and when it is determined that the executable binary has nodependence on the operating system library, using the secondinterception mechanism to intercept file system calls made by the newchild process for accessing a virtual file that does not exist on thefile system arrangement.
 23. The data processing system of claim 22,wherein the intercept library, when in operation, intercepts static ordynamic child binary processes, or both, to provide a virtual filesystem for object storage.
 24. The data processing system of claim 22,wherein the intercept library is arranged to implement the firstinterception mechanism by using LD_PRELOAD.
 25. The data processingsystem of claim 22, wherein the intercept library is arranged toimplement the second interception mechanism by using at least one of: acombination of PTRACE and FUSE, Dynamic Binary Instrumentation, andPTRACE alone.
 26. The data processing system of claim 22, wherein thesecond interception mechanism is chosen based on security or capabilityprivileges available to the parent process.
 27. The data processingsystem of claim 22, wherein the virtual file is a result of conversionof the accessing by the new child process to one or more equivalentobject-storage access operations on a real object located onobject-storage.
 28. The data processing system of claim 22, wherein thesecond interception mechanism is operable to intercept calls made by thenew child process by intercepting file system calls via akernel-provided executable tracing function, wherein the virtual filethat does not exist on the file system arrangement is made accessible tothe data processing system via another virtual file system in a separatepath, and in operation, a system call for opening the virtual file isintercepted, and is firstly handled by ensuring that a virtual filesystem mount is available, and redirecting the system call to a filesaid another virtual file system instead.
 29. The data processing systemof claim 27, wherein the virtual file system mount is implemented as atemporary directory, wherein there is automatically created a mountpoint with the virtual file system so that the mount point is present.30. The data processing system of claim 22, wherein the dynamic linkeremploys, when in operation, the intercept library that intercepts fileaccess operations of the executable software product, wherein thecreation of the new child process preserves the intercept library in anassociated interception environment variable.
 31. The data processingsystem of claim 22, wherein accesses, by the data processing system,under a virtual path are redirected, by the data processing system, to acloud storage arrangement by translating operations to equivalenttranslated requests that are sent over the Internet to the cloud storagearrangement.
 32. The data processing system of claim 22, wherein thedata processing system is operable to provide transparent access ofdata, such that the data processing system uses the intercept library toenable memory-mapped file access operations of an executable to avirtual file, wherein a system call for memory-mapping the virtual fileis intercepted, and handled by first ensuring that another virtual filesystem at a separate mount is available, and redirecting thememory-mapping operation to a file on said another virtual file systeminstead.
 33. A method for operating a data processing system including adata processing arrangement comprising computing hardware, wherein thedata processing arrangement, when in operation, is configured to accessdata from a file system arrangement at least in part by using a dynamiclinker that uses an intercept library to intercept file accessoperations of an executable software product, wherein the interceptlibrary is configured to employ multiple interception mechanisms,including a first interception mechanism to intercept dynamic binariesand a second interception mechanism different from the firstinterception mechanism, wherein the intercept library employs the firstinterception mechanism to intercept a launch by a parent process of anew child process associated with an executable binary, the methodcomprising: intercepting, using an intercept library, the launch of thenew child process by the parent process; determining whether theexecutable binary has a dependence on an operating system library; andwhen it is determined that the executable binary has no dependence onthe operating system library, using the second interception mechanism tointercept file system calls made by the new child process for accessinga virtual file that does not exist on the file system arrangement. 34.The method of claim 33, wherein the intercept library, when inoperation, intercepts static or dynamic child binary processes, or both,to provide a virtual file system for object storage, wherein the objectstorage is accessible by a cloud storage application programminginterface (API).
 35. The method of claim 33, wherein the interceptlibrary is arranged to implement the first interception mechanism byusing LD_PRELOAD.
 36. The method of claim 33, wherein the interceptlibrary is arranged to implement the second interception mechanism byusing at least one of: a combination of PTRACE and FUSE, Dynamic BinaryInstrumentation, and PTRACE alone.
 37. The method of claim 33, whereinthe second interception mechanism is chosen based on security orcapability privileges available to the parent process.
 38. The method ofclaim 33, wherein the virtual file is a result of conversion of theaccessing by the new child process to one or more equivalentobject-storage access operations on a real object located onobject-storage.
 39. At least one non-transitory computer-readablestorage medium storing computer-readable instructions that, whenexecuted by a computerized device, cause the computerized device toperform a method for operating a data processing system including a dataprocessing arrangement which, when in operation, is configured to accessdata from a file system arrangement at least in part by using a dynamiclinker that uses an intercept library to intercept file accessoperations of an executable software product, wherein the interceptlibrary is configured to employ multiple interception mechanisms,including a first interception mechanism to intercept dynamic binariesand a second interception mechanism different from the firstinterception mechanism, wherein the intercept library employs the firstinterception mechanism to intercept a launch by a parent process of anew child process associated with an executable binary, the methodcomprising: intercepting, using an intercept library, the launch of thenew child process by the parent process; determining whether theexecutable binary has a dependence on an operating system library; andwhen it is determined that the executable binary has no dependence onthe operating system library, using the second interception mechanism tointercept file system calls made by the new child process for accessinga virtual file that does not exist on the file system arrangement. 40.The at least one non-transitory computer-readable storage medium ofclaim 39, wherein the intercept library is arranged to implement thefirst interception mechanism by using LD_PRELOAD, and wherein theintercept library is arranged to implement the second interceptionmechanism by using at least one of: a combination of PTRACE and FUSE,Dynamic Binary Instrumentation, and PTRACE alone.
 41. The at least onenon-transitory computer-readable storage medium of claim 39, wherein thevirtual file is a result of conversion of the accessing by the new childprocess to one or more equivalent object-storage access operations on areal object located on object-storage.